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This study is based on a doctoral dissertation completed at 
Michigan State University by the senior author (Stiggins, 1972) 
and was sponsored by the Education 200 Research Council (Research 
Report #7), College of Education, Michigan State University, East 
Lansing, Michigan. 



STABILITY OF SEMANTIC FACTOR STRUCTURE AND CHANGE 
IN CONNOTATIVE MEANING OF EDUCATIONAL CONCEPTS DURING TEACHER TRAINING 
Richard J. Stiggins and Joe L. Byers 
Michigan State University 
A series of eleven concepts, eight of which were systematically presented 
in a teacher training course, were rated by 252 undergraduate education 
majors on fifteen semantic differential scales at the beginning and at the 
end of the course in an attempt to assess changes in the factor structure 
(dimensionality) of semantic space and changes in the connotative meanings 
of the concepts. Scales (bipolar adjectives) selected to represent the tra- 
ditional EPA structure developed by Osgood, were found to be best explained 
by a four factor solution which deviated from the hypothesized structure. 
This structure was found to be stable over time and factor scores for each 
orthogonal factor demonstrated significant gains in directions consistent 
with the goals of teacher training. 



STABILITY OF SEMANTIC FACTOR STRUCTURE AND CHANGE 
IN CONNOTATIVE MEANING OF EDUCATIONAL CONCEPTS DURING TEACHER TRAINING 
Richard J. Stiggins and Joe L. Byers 
Michigan State University 
A prime concern of those dealing with the evaluation of educational pro- 
grams is the creation of new methods of quantifying observation through the 
application of already existing psychometric techniques in new and untested 
ways* The reason for the primacy of this concern is, of course, that new 
uses of observational procedures increases the breadth and scope of the po- 
tential tools available for plotting the influence of an educational program. 
This study deals with an initial exploration of such a novel application in 
the hope that, at some point in the future, it will contribute valuable infor- 
mation for the planning of future 7~struction. 

More specifically, an attempt has been made to assess the influences of 
instruction during teacher training on the affective reactions of trainees to 
the pedagogical concepts, tools and procedures presented in instruction by 
means of the semantic differential scaling procedure (Osgood, Suci and 
Tannenbaum, 1957). Kerlinger (1964) has raised questions concerning the 
stability of the factor structure of semantic space as measured by the seman- 
tic differential (Osgood, et. al., 1957) during teacher training experiences. 
Walberg, et. al. (1968) and Hoover and Schutz (1968) have attempted to deal 
with the question of change in the semantic meaning of concepts as the result 
of teacher training, but the question of changing dimensionality is as yet 
unanswered. The purposes of this study were to (1) assess the factor structure 
of the semantic meaning space of educational concepts to determine if the 
traditional "evaluative," "potency" and "activity" supply; (2) to determine if 
factor structure of ratings remains stable when measured before and after a 
teacher training experience; and (3) to further assess the changes in 
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connotative meaning of concepts as a result of such an experience; in ".he hope 
that the results would lead to meaningful conclusions concerning changes in 
affective reaction to concepts, where affect is operational a as semantic 
differential ratings. Given meaningful conclusion*, a potentially useful 
application of the semantic differential technique would be revealed. 
Instruments 

A semantic differential (S.D.) instrument was developed using eight con- 
cepts systematically presented in a teacher training course in educational 
psychology. These concepts reflected the two phases of the course. One phase 
dealt with the task demands of teaching through presentation of psychological 
concepts related to teaching such as: Reinforcement, Shaping, Respondent 
Learning and Behavioral Objectives. The second phase treated the personal 
demands of teaching by dealing with such concepts as: Non-verbal Behavior, 
Questioning and .Listening Skills, Myself As A Teacher, and Myself. In addition 
to these eight concepts, i^ree were added which were unrelated to Teacher 
Education: Marijuana, Religion, and Physician. These were included in the 
absence of a control group as a quasi control for the purpose of drawing 
some tentative conclusions concerning the cause of changes. 

Each of these concepts was rated on 15 bipolar adjective scales selected 
on the basis of prior research and pilot testing to represent the traditional 
evaluative, potency and activity dimensions of connotative meanings, as defined 
by Osgood, et. al. The scales used were as follows: 

Evaluative Potency Activity 

unfair - fair worthless - valuable relaxed - tense 

bad - good lenient - severe passive - active 

negative - positive weak - powerful insensitive - sensitive 

unimportant - important gentle - violent still - moving 

uninteresting - interesting 
unpleasant - pleasant 
unenjoyable - enjoyable 
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The order and direction of these scales on the S.D, instrument was determined 
by means of a randomization procedure. 

These S.D. elements were combined with instructions adapted from the 
Osgood, et. al. instructions to form the test booklet with one concept per 
page (OpScan response sheet). 
Subjects 

The subjects were 252 undergraduate education majors at Michigan State 
University who were enrolled in the first course in the teacher training 
sequence. This figure represents 81% of the enrollment during the term when 
ratings were gathered. Since part of the affective or connotative meaning 
reactions sought were intended to be of concepts familiar to the respondent, 
subjects were eliminated who had not demonstrated mastery of the denotative 
(definitional) meaning of each concept at the time of final rating. 
Procedures 

The data collection procedures stipulated that the 11 concepts be rated 
on the 15 scales during the first and the last week of. instruction. The time 
interval between collections was approximately nine weeks during which students 
were proceeding through the instructional sequence. The S.D. booklets admin- 
istered at each time were identical except for slight adjustments in the 
instructions to make them appropriate for the time of the rating. 
Analyses 

The purposes of the analyses were to (1) test the hypothesis that a 
three factor solution (evaluative, potency and activity factors) was the most 
parsimonious explanation of the scale interrelationships; (2) test the hypo- 
thesis that this factor solution was stable over time; and (3) that there 
were changes in the connotative meanings of individual concepts within this 
factor structure or frame of reference. 

The first hypothesis was tested by means of an unlimited maximum 
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likelihood factor analysis (Joureskog, 1965) which yields a chi-square test 
of goodness of fit of a factor model, Cattel f s (1966) Scree test based on a 
graphic representation of latent roots, and by means of a rational analysis 
of the factor loading matrices. The correlation matrix of scale ratings was 
factor analyzed separately for each concept on both the pretest and the post- 
test. However, due to the similarities among these 22 factor solutions and 
due to the complexity of interpreting 22 solutions, the correlation matrix 
which resulted from pooling concept ratings over pretest and posttest were 
also factor analyzed. The two resulting factor solutions are reported and 
dicusssed below. 

The second hypothesis, concerning factor structure invariance, was 
assessed by means of a Pearson product moment correlation coefficient, cor- 
relating factor loadings of scales on comparable factors from the pretest 
and the posttest factor loading matrices, as suggested by Cattell (1966). 

The test of the final hypothesis of change in individual concept meaning 
was based on pretest and posttest factor scores generated by employing factor 
loadings as regression weights and computing a sum of weighted ratings in 
accordance with the procedures and formulation suggested by Harman (1960): 

Y = XS" l ji 

where Y is the matrix of factor standard scores, X is the matrix of variable 
scores, S is the inverse of the variance-covariance matrix, and A is the 
varimax rotated factor loading matrix. This computational procedure resulted 
in a factor score for each concept on each orthogonal factor for both pretest 
and posttest ratings. In order to test the hypothesis of alteration in 
concept meaning, changes in factor scores (post minus pre) for each concept 
served as multiple dependent measures in a one factor one level multivariate 
analysis of variance designed to assess the difference of each change from 
zero. This test was carried out separately for each orthogonal factor. 
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Results 

Factor Structure 

It was clear by all criteria (X 2 test, Scree test, and factor loading 
•patterns) that a three factor solution was inappropriate. Space limitations 
do not permit presentation of all results. A complete analysis may be found 
in Stiggins (1972). However, Table 1 does report the results of the X 2 test 
of goodness of fit of the three factor model. Since the hypothesized structure 
was appropriate in only two of the 22 factor analyses, 



Insert Table 1 about here 



it was concluded that a more complex latent structure was operating and further 
analyses were carried out to discover its dimensions. 

Extensive manipulation of the pooled and unpooled scale correlation 
i«atrices and a careful consideration of all of the criteria listed above 
revealed that the most parsimonious factor solution, from both a statistical 
and a rational point of view, was a four factor model. Once again, space 
limitations do not allow for a complete representation of the data} However, 
the pretest and posttest varimax solutions reported in Table 2 provide evidence 
of the latent factors or dimensions of connotative meaning which were tapped 
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by the scales selected. These factor loadings suggest that the scales were 
interrelated in such a way as to represent the following dimensions: 



Stiggins (1972) for complete description of the factor solutie 
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Meaning 
Dimension 

Evaluative 



Personal 
Evaluative 



Characteristic 
Scales 

worthless - valuable 
bad - good 
negative - positive 
unimportant - important 



unenjoyable - enjoyable 
unpleasant - pleasant 
(uninteresting - 
interesting) 



Leniency 



Potency 



severe - lenient 
tense - relaxed 
violent - gentle 
(insensitive - 

sensitive) 
(unfair - fair) 

active - passive 
still - moving 
weak - powerful 



Meaning Definition 

This dimension represents 
the respondent's favorable 
or unfavorable reaction to 
the concept in terms of 
how it affects others. 

This dimension reflects 
the respondent's favorable 
or unfavorable reaction in 
terms of his own personal 
values or from the point 
of view of how it affects 
him. 

These scales reflect the 
meaning of the concept 
from a flexibility point 
of view or on an open vs. 
closed dimension. 



This factor represents a 
reflection of the manifest 
or observable power and 
activity potential of each 
concept rated. 



Factor Structure Invariance 

As reported in Table 3, there was a very high degree of stability in the 
factor structure as demonstrated by the high correlations between loadings on 
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comparable pretest and posttest factors. It is inferred from this that 
respondents employed the scales to relate the same dimensions of meaning 
at the beginning and at the end of instruction. It was then poss. e to 
assess changes in the meanings of concepts within these stable factors or 
dimensions of meaning. 

Changes In The Connotative Meaning Of Concepts 

The factor loading matrices reported in Table 2 were combined with the 
raw responses in the manner prescribed by Harman (1960) to yield an index of 
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an individual's favorable or unfavorable reaction to each concept (evaluative 
dimension) at the beginning and at the end of the term, how he reacted to it 
on the personal evaluative (pleasant - unpleasant) dimension, his reaction in 
terms of its leniency, and the potency he perceived in each concept. The 
change in each of these reactions over the terras of instruction, defined as 
the difference between the factor scores (post minus pre), was then assessed 
on each dimension. 

Tables 4-7 report the MANOVA 1 s of change on each of the four dimensions. 
It is apparent on the basis of the multivariate F f s that tnere were significant 

Insert Table 4 about here 

overall changes on the Evaluative (multivariate F = 28.43, p < .0001), 
Leniency (multivariate F » 4.98, p < .0001), and Potency (multivariate 
F = 8.24, p < ,0001) dimensions, but not on the Personal Evaluative (multi- 
variate F = 1.74, p < .0651) dimension. Further, it is apparent that changes 
occurred in the instructional concepts which did not occur in the noninstruc- 
tional concept group. Evidencu of this is seen in the Stap Down F ! s in Tables 4 
Reading from the bottom of that column, the three noninstructional concept 
mean changes must be considered significant, however, the univariate F's 
temper this slightly. On the Evaluative dimension (Table 4) Questioning 
and Listening Skills and Reinforcement show minimal change. Small changes 
in the Potency dimension (Table 5) are seen for Myself and, on the Leniency 
(Table 6), Non-verbal Behavior, Myself, Behavioral Objectives and Reinforce- 
ment changed a little. 

However, these change data supply only part of the picture provided by 
the data. The factor score means also provided information for a geometrical 
interpretation of the connotative meaning ascribed to each concept (Osgood, 
et. al., 1957). For example, by using the orthogonal factor scores as vectors 
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in Euclidean space, one is able to construct an N dimensional semantic mean- 
ing space, where N is the number of orthogonal factors. In this particular 
case, four orthogonal factors were extracted. However, there were significant 
changes on only three of these. Consequently, a three dimeasional space was 
constructed to reflect the connotative meaning of a concept with respect to 
the Evaluative dimension (favorable - unfavorable), the Potency dimension 
(potent - impotent) and the Leniency dimension (severe - lenient). The mean 
factor score (over subjects) for each dimension, both pre and post, as reported 
in Table 8, was then plotted on the three orthogonal axes. The location of 
each concept in this semantic meaning space is reported in Figures 1-3. 
Each of the three concept categories is reported separately to add clarity 

Insert Figures 1 - 3 & Table I about here 

to their relative positions. In each case the arrow indicates the change 
from pre to post. Since no concept was rated oevere, the figures include 
only the lenient half of the lenient - severe/ favorable - unfavorable principle 
plane. 

Discussion 

Scale Interrelationships and Factor Analysis 

The finding that the Osgood EPA stricture was inappropriate when scales 
had been selected to reflect those dimensions reinforces two points made in 
prior thinking concerning S.D. research. The result (i.e., unexpected four 
factor solution) reinforces the validity of the Smith (1961) and Heise (1969) 
argument that the dimensionality of semantic space nust be reassessed for 
each new application of the technique. In addition, it supports the Wittrock 
(1964) finding the EPA structure, though appropriate for the general meaning 
domain, does not seem to apply to concepts from Teacher Education. And finally, 
this result reveals potential errors which can be made in S.D. research by 
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choosing scales on the basis of prior research and to generating factor sco 
without verifying that the desired latent structure has in fact been tapped 

Some discussion of the four factor latent structv~_ ^eeras warranted. 
Evaluative dimension is a manifestation of the dimension of meaning tapped 
most frequently and which regularly accounts for the largest amount of comm 
variance in S.D. research. It is interesting to note, however, that a seco 
factor of an evaluative nature has been tapped. This is the Personal Evalu 
tive factor. Though there is a conceptual similarity between favorable and 
unfavorable reactions in terms of how a concept affects others and a favora) 
to unfavorable reaction using one's self as the frame of reference, there a 
indications that these are independent judgments. That is, i*. would seem 
that the judgment as to the value of a pedagogical tool is made somewhat 
independently of person&l like/Uislike considerations. The remaining two 
factors, leniency and potency, appear to be a breakdown and reassembly of 
Osgood's potency and activity dimensions. For example, thr scale weak - 
powerful is most closely associated with elements of the Osgood activity 
dimension (active - passive), and violent - gentle which characterizes Osgo< 
potency dimension is associated with tense - relaxed which has been cousidei 
an element of the activity scale in the Osgood research. These variations j 
factor structure deserve further exploration and verification in future res* 

The question of the stability of the latent structure requires little 
discussion, except that the high degree of Invariance sheds some light on tl 
question posed earlier by Kerlinger. It would seem that, in this particul&i 
case, this component of teacher training experience has Httle influence on 
the dimensionality of the educational semantic space of the trainee. 
Individual Concept Meaning, Change 

It was stipulated that if instructional concepts changed in meaning and 
the noninstructional did not change, some tentative inferences as to the 
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cause of the change would be possible. Since this was in fact found to be 
the case, the tentative suggestion that the instruction did contribute to the 
change seems warranted. However, more convincing evidence could have been 
obtained with a slight alteration in the concepts employed. If a category of 
general educational concepts (not presented in instruction) had been rated, 
there would have been information available on which to decide if instruction 
had influenced only instructionally relevant concepts or had merely brought 
about gross changes in attitude toward education which would be reflected in 
any educational concept. On the basis of the data presented here, such a 
gross alteration in emotive responses cannot be ruled out as a plausible rival 
explanation for the results. Further research must clarify this point if the 
S.D. procedure is to supply useful information in the evaluation of specific 
courses and course content. 

The relative magnitudes of the changes which took place in the different 
concepts suggest an interesting relationship between the initial state of 
connotative meaning and the potential for changing that meaning. The concepts 
which showed greatest change were Behavioral Objectives, Respondent Learning 
and Shaping. The prior meaningful associations or experience which respond- 
ents had with these tern-* are probably minimal in relation to the prior 
experience they had had with such concepts as Myself, Non-verbal Behavior or 
Questioning and Listening Skills. This suggests that there is an inverse 
relationship between familiarity with a concept and the amount of change in 
connotative meaning which can be achieved in a person's reactions to the 
concept Jwheh the amount of energy (i.e., instruction) expended in bringing 
about such a change is held constant. This relationship deserves further 
exploration, because it may serve as an important link between denotative 
meaning (cognitive) and connotative meaning (affective). 
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The locations of the concepts in the three dimensional space are also 
of interest. First, the widely different meanings of the noninstructional 
concept and the location of marijuana in the unfavorable domain add credibil- 
ity to the psychometric technique, because these might be expected from a 
rational point of view. The two clusters of instructional concepts (self vs. 
procedural) also lend some support to the validity of the measures, because 
one might expect these to exist in the domain of connotative meaning (i.e., 
the self more lenient than tools employed in the classroom). This research 
would probably be classified as dealing more with the tools, concepts and 
procedures than with the self. But, perhaps this instrumentation or technique 
could be profitably applied to self consideration in more detail, as in 
measuring changes in individual perceptions of self and others. 

The question of the actual connotative meaning of the concepts (location 
in semantic space) and the changes therein lead directly to the discussion 
of the appropriateness of these meanings for teacher training. It would seem 
that, since each of the instructional concepts were seen as favorable, potent 
and lenient, students are predisposed to use them in constructing a learning 
environment. Further, since they were seen as more favorable, potent and 
lenient after instruction, the influence of instruction was to increase the 
predisposition. However, it should be made clear that this may be reading 
too much into the data. There is no priof that scores ca S.D. rating of 
educational concept are predictive of later classroom behavior. Though there 
is no evidence that S.D, scores are predictive of be,i&vior in ether areas 
such as empathetic responses in counselors (Greenberg, 1970) and movement 
toward mental health (Endler, 1961), the assumption that the same relation- 
ship exists when classroom behavior is the criterion, may be invalid. The 
research reported here is proceeding on such an assumption for purposes of 
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developing the assessment technique and procedures, but the research cannot 
proceed much further until this validation study is carried out. 

Cone lusions 

On the basis of a pretest - posttest administration of a semantic dif- 
ferential instrument on which undergraduate education majors rated eight 
concept tools and procedures relevant to instruction in teacher training, the 
following conclusions seem warranted: 

1. For this particular set of educational concepts, the traditional 
evaluative, potency and activity dimensions are inappropriate, but there is 
a fairly clear four latent structure which seems to characterize the scale 
in terrela t ionships . 

2. This four factor solution is highly stable over the nine week period 
of teacher training which served as the treatment in this study. 

3. There were discernible changes in the connotative meanings of the 
tool concepts and procedures rated on three of the four meaning dimensions 
tapped, which result in the concepts being rated as more favorable, lenient 
and potent at the conclusion of instruction than at the outset of instruction. 

The issues that remain unsettled are the reproducibility of the factor 
structure, the inference of a cause and effect relationship between instruc- 
tion and change in connotative meaning, and, most importantly, the relation- 
ship between these S.D. ratings and the actual classroom behavior of the 
teacher. Any statement as to the future of the procedures employed here as 
a valuable evaluation technique in teacher training must await resolution of 
these issues. However, the research reported here would appear to be a step 
in a fruitful direction. 
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Table 1: Fit of the three factor model to the pretest a.ad 
posttest correlation matricies 



Concept Pretest 




MYSELF AS A TEACHER 


X 2 » 93.5 


X 2 = 81.8 


NONVERBAL BEHAVIOR 


154 .4 


136.9 


QUESTIONING AND LISTENING SKILLS 


99.4 


106.5 


MYSELF 


152.9 


166.7 


BEHAVIORAL OBJECTIVES 


134.2 


170.2 


REINFORCEMENT 


135.7 


152.1 


RESPONDENT LEARNING 


219.2 


144.5 


SHAPING 


236.3 


139.6 


PHYSICIAN 


132.1 


78.1 


RELIGION 


157.7 


261.5 



a The null hypothesis that the model does not fit the 
data is rejected at x 2 - 82.22 where a = .05 and df = 63. 

Three-factor model fits- 
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Table 3: Pearson Product Moment Correlation 

Coefficients between Factor Loadings 
on each of the Four Meaning Dimensions 



Meaning 


Correlation 


Dimension 


Coefficient 


Evaluative 


,998* 


Personal Evaluative 


.993* 


Leniency 


.887* 


Potency 


.891* 



* p < .01 
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• 1748 


14.6749 




.0002 


---- — ■ 

14.G749 


,0002 


Nonverbal Behavior 


.3288 


30.3106 


.0001 


26.2311 


.0001 


Questioning and 
Listening Skills 


-.0594 


1 .2661 


• 2 616 


1.8232 


1 7 A 9 


Myself 


.2376 


28 .7822 


• u u U 1 


ID • £. J ->tJ 


. v» 0 0 1 


Behavioral Objectives 


.574 ^ 


57.2381 


.0001 


46.893? 


.0001 


Reinforcement 




A O C O A 


.0283 


1 . 22 54 


, ,2694 


Respondent Learning 


.6662 


89*0564 


.0000 


27.6389 


.0001 


Shaping 


1.0439 


205.0043 


.0000 


87.6834 


,0000 


Physician 


. 2826 


14 .4736 


.0002 


0.5649 


.4531 


Rel ig ion 


.1295 


4.3497 


.0381 


0,9137 


,3401 


Marijuana 


.1762 


7,3671 


,0072 


1.3500 
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LENIENCY DIMENSION 












Myself as a Teacher 


.1032 


5.6118 


.0136 


5.6118 


,0186 


Nonverbal Behavior 


.0567 


0*7388 


,3909 


0.0209 


,8852 


Questioning and 
Listening Skills 


.1768 


7 ,9529 


,00 52 




, UUOl 


Myself 


.0682 


1 .7833 


, 18 30 
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, o JO b 


Kehavioral Objectives 


• 0186 


0.0659 


.79 75 






Reinforcement 


-.0247 


0.1224 
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Respondent Learning 
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Shaping 
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Physician 
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0.2161 


.6425 


0,5677 


,4519 


Religion 


-.0341 


0.2038 


.6521 


0,3195 


,5725 


Marijuana 


• 0166 


0.0530 


.8182 


0,0608 


,8055 



POTENCY DIMENSION 

Myself as a Teacher 

Nonverbal Behavior 

Questioning and 
Listening Skills 

Myself 

Behavioral Objectives 

Reinforcement 

Respondent Learning 

Shaping 

Physician 

Religion 

Marijuana 



.2739 
.2594 

.2426 
.1766 
.4383 
.3536 
.4112 
.4318 
.2606 
.1841 
.0440 



30,4171 
10.6794 

12.1528 
8.7039 
34.3504 
26.3572 
36.1644 
38.0995 
15.4934 
5.7773 
0.3373 



• 0001 
.0013 

.0006 

• 0035 

• 0001 

• 0001 
.0001 

• 0001 

• 0002 
.0170 
.5620 



30.4171 
4.7458 

3.2651 
1.6756 
14.7340 
9.1825 
6.8243 
7.4633 
1.6291 
0.5850 
1.1745 



.0001 
.0304 

.0720 
.1968 
.0002 
.0028 
.0096 
.0068 
.2031 
.4451 
,2796 



PERSONAL EVALUATIVE DIMENSION' 



ERLC 



Myself as a Teacher .1215 4.6531 

Nonverbal Behavior .0409 0.2629 
Questioning and 

Listening Skills .1191 1.8549 

Myself ,1544 6.2616 

Behavioral Objectives -,0739 0,8365 

Reinforcement ,0C19 0,0006 

Respondent Learninu .0197 0,0646 

Shaping ,i5 ? 8 4.0215 

Ph>sicun -.1269 2.3823 

'\C 1 1 1 on - 1 *5 t\ \ a r a 



.0320 
.6086 

.1745 
.0130 
.3613 
.9809 
.7996 
.0460 
.1240 



4.6531 
0,2241 

1,7434 
4 ,6452 
1.4277 
0,1368 
0,0362 
3.6331 
2 .5557 



,0320 
,6361 

,1880 
,0322 
,2333 
.7118 
,8458 
.0579 
,1112 



Table 4: Results of Multivariate Analyses of Variance of Change 
in the Connotative Meaning of Concepts on each Meaning 
Dimension. 
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